TutorNet: Towards Flexible Knowledge Distillation for End-to-End Speech Recognition

In recent years, there has been a great deal of research in developing end-to-end speech recognition models, which enable simplifying the traditional pipeline and achieving promising results. Despite their remarkable performance improvements, models typically require expensive computational cost to show successful performance. To reduce this burden, knowledge distillation (KD), is popular model compression method, used transfer from deep complex (teacher) shallower simpler (student). Previous KD approaches have commonly designed architecture student by reducing width per layer or number layers teacher model. This structural reduction scheme might limit flexibility selection since structure should be similar that given teacher. cope with limitation, we propose new method for recognition, namely TutorNet, can across different types neural networks at hidden representation-level as well output-level. For concrete realizations, firstly apply (RKD) during initialization step, then softmax-level (SKD) combined original task learning. When trained RKD, make use frame weighting points out frames pays more attention. Through experiments on LibriSpeech dataset, it verified proposed not only distills between topologies but also significantly contributes improving word error rate (WER) distilled student. Interestingly, TutorNet allows surpass its teacher's some particular cases.